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METHOD OF SEARCHING MULTIMEDIA DATA 

BACKGROUND OF THE INVENTION 
Field of the Invention 
5 The present invention relates to a method of searching 

multimedia data, and more particularly to a method of searching 
multimedia data more accurately by utilizing user feedback. 
Background of the Related Art 

Recently, technology for digital image signal processing has 
So been greatly developed and has been applied in various fields. 

For example, the digital image signal processing technology may 

= 

be used in a search system for automatically editing only the 
face of a specific character in a moving picture file of a movie 

M drama, in a security system for permitting access to only 

its persons registered in the system, or in a search system for 
searching a particular data from an image or video detected by 
a detection system. The performance of such systems basically 
depend on the accuracy and speed of detecting or searching the 
desired object. Accordingly, various image searching methods have 

20 been proposed in the related art. 

An image search system which detects a degree of similarity 
with an image to be searched utilizing features such as color, 
texture or shape is disclosed in USPAT 5,579,471 entitled "An 
image query system and method*'* Depending upon the image to be 

25 searched, the importance of a feature may vary and within one 
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particular feature such as the color, the importance of a feature 
element such as the red or green color may also vary. However, 
the above searching system does not take into consideration the 
different importance of features or feature elements for each 
5 image to be searched. 

In another searching method entitled "Virage image search 
engine" (www.virage.com), a user direcrly inputs the level of 
importance for features such as the color, texture and shape by 
C3 assigning weight values. Although an image may be searched 

JgO according to an importance of a feature using this method, it may 
if^ be difficult for a user to determine the weights of features. 
|5 Therefore, Yong Rui in "Relevance feedback techniques in 

interactive" SPIE Vol. 3312, discloses a method in which images 
U similar to a reference image are found and the importance of 
fli£ features or weight for features are automatically obtained by 
n calculating the similarities among the found images. However, the 
weight importance information is not maintained after a search 
for a specific image is finished and must be calculated for each 
image search, even for a same image. 
20 In the image search and browsing system or the video (moving 

image) search and browsing system of the related art, information 
which describes a particular feature of an image or video data 
is utilized to perform a more effective search or browsing of the 
multimedia data. For example, in the image query system, an image 
25 may be divided into a plurality of regions and a representative 
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color of each region may be utilized as a feature information of 
the image, or a whole color histogram of the image may be 
utilized as a feature infprmarion. Thereafter, two images are 
compared ro calculate a similarity based upon the feature 
5 information and a determination is made whether the two images 
are similar. 

Therefore, the image search methods in the related art may 
utilize weights of features such as color, texrure, or shape. 
C3 However, weights of feature elements .are not raken into 

19 consideration. Accordingly, the image search methods in related 
[fi art has the limitations in intellectually training and developing 

weights for searching, resulting in relatively longer searching 
period to obtain search results and a deterioration of the 
reliability of the search results. 

ly 

13 SUMMARY OF THE INVENTION 

f ^ 

Accordingly, an object of the present invention is to solve 
at least the problems and disadvantages of the related art. 

An object of the present invention is to provide a method 

20 of searching multimedia data by automatically updating weights 
of features included in a specified image and/or weights of 
feature elements, and by applying the updated weights to search 
for the specified object. 

Another object of the present invention is to provide a 
25 method of searching multimedia data which constructs image 

3 
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characteristics corresponding to the types of features included 
in a specified image by analyzing and classifying the judgement 
standards applied when the user searches the image, and adjusts 
the feature information set by talcing into consideration weights 
5 of the features and weights of feature elements during a 
following image search. 

Still another object of the present invention is to provide 
a feature structure to be included in a multimedia data to 
effectively search an image, 
ifi Additional advantages, objects, and features of the 

■'^ invention will be set forth in part in the description which 
l'^ follows and in part will become apparent to those having ordinary 
13 skill in the art upon examination of the following or may be 
C3 learned from practice of the invention. The objects and 
i§ advantages of the invention may be realized and attained as 

£ r 

particularly pointed out in the appended claims. 

'"'^ To achieve the objects and in accordance with the purposes 

of the invention, as embodied and broadly described herein, a 
method of searching multimedia data in a multimedia data search 

20 system comprises searching for a reference multimedia data 
selected by a user; receiving user feedback of relevance 
information for the searched multimedia data; determining 
importance of respective feature elements of features included 
in the multimedia data according to the relevance information; 

25 re-performing the search for the reference multimedia data by 

4 
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updating the importance of said respective elements if the user 
requests an additional search; and updating previous importance 
to new importance obtained and maintaining the updated importance 
degrees . 

5 In another embodiment of the present invention, a method of 

searching multimedia data in a multimedia data search system 
comprises receiving an inquiry into previously searched 
multimedia data; analyzing a judgement standard for the 
C3 multimedia data searched during the inquiry; constructing image 
IJ) characteristics using at least one feature included in the 
In multimedia data using an analysis result of the judgement 
1^ standard; and adjusting importance of the image characteristics 
and re-performing a search of the multimedia data if a user 
requests an additional search. 
a The present invention also provides a feature st^ructure of 

|3 multimedia data comprising a first information representing a 

r ^ 

feature of the multimedia data; a second information representing 
a regional feature of the multimedia data; and a third 
information representing importance of the first and second 
20 information. 

BRIEF DESCRIPTION OF THE DRAWINGS 
The invention will be described in detail with reference to 
the following drawings in which like reference numerals refer to 
25 like elements wherein: 
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FIG. 1 Shows features in an image by a histogram; 

FIG, 2 shows an image represented by a local grids; 

FIG. 3 is a flowchart of a multimedia data search process 
according to a first embodiment of the present invention; 
5 FIGS. 4 and 5 are flowcharts of a multimedia data search 

process of FIG, 2, where an initial search does not result in a 
desired image; 

FIG. 6 is a basic structure of image characteristics for an 
[3 image search according to a second embodiment of the present 

.50 invention; 

in F1G\ 1 is image characteristics of FIG. 6 wherein the 

\% feature information is constructed using a color and texture; and 

FIGS. 8 to 11 are different embodiments of the image 
\^ characteristics , 

as 

£3 DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

Reference will now be made in detail to the preferred 
embodiments of the present invention^ examples of which are 
illustrated in the accompanying drawings. 

20 In a general image search system, features of an image such 

as color, texture, and shape may be represented by a histogram 
as shown in FIG. 1. Particularly, FIG. 1 is a color histogram of 
an image by which the colors in the image is grouped into 2 4 
color elements. By adjusting or altering the weights of each 

25 color element, the degree of importance of each color element or 
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rhe extent that a color element affects a search may be 
represented. 

FIG. 2 is an image represented by local grids. Particularly, 
an image is divided into n'^'m grid regions and each grid region 
5 may be represented either by a regional color histogram or by a 
color representing the grid region. Here, the degree of 
importance of each grid region or the extent that a grid region 
affects a search may be represented by assigning weights to each 

Q 

^3 of n^'m grid regions. Also, a predetermined threshold value 
J4) determined by the system may be utilized as a cap such that 
l:\ certain grid regions has no influence in the search. Namely, if 
^ij a grid region has a weight which exceeds the predetermined 
threshold value, the grid region is utilized in the search. 

r ~i 

H Otherwise, the grid region is processed as a "Don't care" region 

^3 which does not affect the image search. 

f ^ 

n The image search process will next be explained with 

reference to FIGS. 3 to 5 . 

FIG. 3 is a flowchart of a multimedia data search process 
when a plurality of reference images are selected and weights of 

20 the respective types or elements of features included in a 
specified image are assigned or updated according to the first 
embodiment of the present invention. Referring to FIG. 3, a user 
selects a plurality of reference images (step S301) representing 
the specified image to be searched. Thereafter, the system judges 

25 and determines the importance of the feature elements included 
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in the reference images. 

For example, the eystem measures the similarities of the 
features included in the reference images (step S302), and 
determines the weights of each feature according to the measured 
5 similarities of the features (step S303). Also, the system 
measures the similarities of the feature elements in each feature 
included in the reference images (step S304), and determines the 
weights of feature elements in each respective feature according 
[1 to the measured similarities of the feature elements (step S305) . 
^1 Accordingly, if a user requests an additional search, the 

system re-performs the image search by using the importance, i.e. 
the weights of the features and the weights of the feature 
elements in the respective features (step S306) . At this time, 
p the system may use either the weights of the features or the 

4i weights of the feature elements, or both. 

I™ 

n Particularly, the weights of the features and feature 

'^^ elements of respective features are determined as follows • When 
the user selects a plurality of the reference images, the search 
system determines weights of the features and the weights of the 
20 feature elements in the respective features by measuring the 
similarities of features of the selected reference image list and 
the similarities of the feature elements in the respective 
features, The weights of the features are calculated by Equations 
la, lb , and Ic below. 
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tla] Weight k^_C£!im_ 

Conl(ALL) 



n-l n 



[lb] Cow/(/r) = 2^ J^^w^y.^) 



lie] Coni{ALL) = £ Contik) 

In the above equations, n denotes the reference number, m 
^5 denotes the number of features used for measuring the similarity, 
We±ght_k denotes the weight of the k-th feature, Slmx ,k) 
Vl denotes the similarity between the I-th reference image and the 
In j-th reference image when the k-th feature is used, and Cont (k) 

o 

. denotes how much the k-th feature contributes to raise the 

\Q similarity. Generally, the weights of the respective features 
H Welght_k increases as the similarities rises since the 
m similarities among the reference images are calculated based upon 

the respective features and the feature having the highest 

similarity acts as the most important factor. 
15 The weights of the feature elements are determined by 

Equations 2a and 2b using the similarities of the feature 

elements in the respective features of the reference images. 



[2a] Weight of element i: w^^afjii) 
20 [2b] Similarlity of element i in reference images 



9 



2000-JAN-31 13:57 FROM: KIMS INTERNATIONAL 0E-557-340^ 

# 



In rhe above equations, the values of a, p, and q denote 
constants, jti^ denotes an average of an element i in the reference 
image list, and denotes the distribution of an element i in 
5 the reference image list. According to Equations 2a and 2b, the 
weight Wi of a feature element is inversely proportional to the 
C3 distribution of the corresponding feature element and is 

^1 proportional to an average value of the corresponding feature 
\f\ element component. Therefore, a feature element having a large 

rij 

J^O average value acts as an important factor even if the 
distribution of the corresponding feature element is wide. 

i 

H FIG. 4 is a flowchart showing a multimedia data search 

process for when a plurality of reference images are and are not 

L J. 

[3 selected. Generally, if the number of reference images is not 
15 plural, the image search is performed using features with 
equivalent weights. However, if the search does not result in a 
desired image, other reference images similar to the specified 
image to be searched are selected and added to a reference image 
list* Then, the weights of the respective features and/or feature 
20 elements are updated using the reference image list. 

Referring to FIG, 4, when a user selects a plurality of 
reference images (steps S401 and S402), the process (steps 
S404'-S408) is the same as described with reference to steps 
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3302-3306 of FIG. 3. Thus, an explanation will be omitned. 
Thereafter, if the user is satisfied with the result of the image 
search (step S408), the search operation terminates. However, if 
the user is not satisfied with the search result, the user 
5 selects other reference image (s) similar to the specified image 
to be searched among the resultant images of the search (step 
S411) . On the other hand, when a user selects only one reference 
image, the image search is performed using features with weights 
C3 as assigned (step S403) . Generally, if the reference image is 
;E0 selected for the first time, the search is performed using 

[n features with equal weights assigned. If the user is not 

fii 

jS satisfied with the search result, the user selects other 

reference image (s) similar to the specified image to be searched 
among the resultant images of the search (step S411> . 

CILB Accordingly, the system adds the selected other reference 

CD image (s) to an initial reference image list managed by the system 

(step S412) . Here, the initial reference image list includes the 
reference image (s) selected in step S401. Thereafter, the system 
measures the similarities of the respective features and/or 

20 feature elements in the selected reference images (step S413) , 
and determines the weights of the respective features and/or 
feature elements in the selected reference image by Equations la 
- Ic, 2a and 2b (step 3414) . Thus, the system updates the weights 
of the respective features and/or feature elements, and re- 

25 performs the image search utilizing the updated weights (step 
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S415) . 

FIG. 5 is a flowchart showing another multimedia data search 
process for when a plurality of reference images are and are not 
selected. Generally, if the number of reference images is not 
5 plural, the image search is performed using features with 
equivalent weights. However, if the search does not result in a 
desired image, other reference images, both similar and 
dissimilar to the specified image to be searched are selected and 
1=: respectively added to a reference im.age list or a dissimilar 
ii) image list. Then, nhe weights of the respective features and/or 
feature elements are updated using the reference image list and 
the dissimilar image list. 
[3 Referring to FIG. 5, when a user selects a plurality of 

n reference images (steps S501 and S502), the process (steps 
is S504-S508) is same as described with reference to steps S302-S306 

12 of FIG, 3. Thus, an explanation will be omitted. Thereafter, if 

U 

^- the user is satisfied with the result of the image search (step 
S509) , the search operation terminates. However, if the user is 
not satisfied with the search result, the user selects other 

20 reference image (s) similar and dissimilar to the specified image 
to be searched among the resultant images of the search. On the 
other hand, when a user selects only one reference image, the 
image search is performed using features with weights as assigned 
(step S503). Here, if the reference image is selected for the 

25 first time, the search is performed using features with equal 

12 
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weights assigned. If the user is not satisfied with the search 
result, the user then selects other reference image (s) similar 
and dissimilar to the specified image to be searched among the 
resultant images of the search. 
, 5 Accordingly, the system adds the selected similar image (s) 

to an initial reference image list managed by the system {step 
S511) and adds the dissimilar image (s) to an initial dissimilar 
image list (step Sol2) . Here, the initial reference image list 
tj includes the reference image (s) selected in step S501. 
Thereafter, the system measures the similarities of the 

%J 

[n respective features and/or feature elements in the images 

rii 

;5 included in the reference image list (step SS13), and measures 
the similarities of the respective features and/or feature 
'j^ elements in the images included in the dissimilar image list 
Is (step S514) . 

□ Using Equations la - Ic, 2a and 2b, the system determines 

the weights of the respective features and/or feature elements 
using the images included in the reference image list and the 
dissimilar image list (step S515) . Thus, the system updates the 

20 weights of the respective features and/or feature elements, and 
re-performs the image search utilizing the updated weights (step 
S516) . Particularly, the weights of the features by the 
similarity measurement of the features and/or feature elements 
in the images included in the reference/dissimilar image lists 

25 are calculated by Equations 3a 3d. 

13 
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n 



[3a] Weight _ k = (ax Weight j /, ) x ( ^ ^) 

Weighty t 



[ 3b] Weighlj, , = (Weight, ,) = fZ^^^r), 

Com {ALL) 

[3c] Con(ik) = 2 ^SimiiJ.k) 



rn 

[3d] Conl(ALL) ^^Coni(k) 



S In the above equations^ n denotes the reference number in 

the reference image list or the dissimilar image list, m denotes 
the number of features used for the similarity measurement, 
12 We±ght_k denotes the final weight of the k-th feature, 
p Simla, j,k) denotes the similarity between the I-th reference 
X§ image and the j-th reference image when the k-th feature is used, 
p Cont(k) denotes how much the k^th feature contribute to raise the 
similarity, W^ight^_^ denotes the weight of the k-th feature in 
the reference image list, and Weight^^^ denotes the weight of the 
k-th feature in the dissimilar image list. Generally, the 
15 similarities of the images included in the reference image list 
and the dissimilar image list are calculated respectively. As a 
result, the weights of the respective features Wexght_k increase 
as the similarities of the images included in the reference image 
list rises, while the weights decrease as the similarities in the 
20 images included in the dissimilar image list rises. 

14 
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Also, after measuring the similarities of the feature 
elements in the respective features of the images included in the 
reference/dissimilar image list, the weights of the feature 
elements of the respective features are determined by Equations 
-5 4a 4b, 

[4a3 Weight of an element i: u/^=afj{l)-^hfp^{i) 

[4b] Similarity of an element 1 in the reference images: 

n J An- 

in In the above equations, (I) ~pmj^xv_i denotes the 

l^Q dissimilarity of an element i in the images included in the 
^ dissimilar image list, the values of a, b/ p, and g denote 

constants, jn^ denotes an average of the element i in the images 

11 included in the corresponding (reference and dissimilar) image 

Q 

:^ lists, and denotes the distribution of the element i in the 
15 images included in the corresponding (reference and dissimilar) 
image lists, Generally, the similarities of the images included 
in the reference image list and the dissimilar image list are 
calculated respectively. As a result, the weights of the 
respective feature elements Weight_k increase as the similarities 
20 of the images included in the reference image list rises, while 
the weights decrease as the similarities in the images included 
in the dissimilar image list rises. 

If the weights of the features and the weights of the 

15 
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feature elements are determined as above, the similarities will 
be calculated using Equation 5 during the image search. 

n km 

[5] A-Y^W,Y^w,jDiMFuj,P,q) 

Here, ^ is a constant, Diff(F^xP^^) denotes the difference 
5 between the I-th elements of the k-th feature of the image p and 
image g, t^.. j denotes the weight of the I-th feature element of 
P the k-th feature, w^, denotes the weight of the k-th feature, n 
denotes the number of features, and km denotes the number of 

'\2 

feature elements of the k-th feature. Thus, the difference is 

f 1 \ 

:|S obtained by multiplying the feature difference value of the 
respective image, the feature element weight of the respective 
P feature, and the weight of the respective feature. Also, the 
|y similarity is obtained by subtracting the difference from the 
n constant. 

rS As described above in reference to FIGS. 3-5, the system 

automatically determines and updates both the feature element 
weights of respective features and the weights of the features 
of the image to be searched when the user searches an image. 
Therefore, a rapid and effective search can be performed. 

20 Nevertheless, if the user wishes to perform a further search 

of the specified image after viewing a previously searched 
result, the user may raise and enter various kinds of queries to 
the search system. Table 1 shows examples of queries by users, 
and Table 2 shows the feature information required according to 

16 
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the type of query when colors and textures are used as the basic 
features of an image. 







Table 1 


5 




Query Type 




1 


What color does image have as a whole? 




2 


Does anv Dortion of the imacre have a certain color 

f earure? 


U 


3 


About: what degree does the portion of the image have a 
certain color feature? 




4 


What texture does the image as a whole? 


iB 


5 


Does any portion of the image have a certain texture 
feature ? 


f ^ 


6 


About what degree does the portion of the image have a 
certain texture feature? 


% 


7 


Does the image have a certain color and texture 
feature or have any portion having such features? 




8 


Does any portion of the image have a certain color and 
texture feature? 




9 


About what degree does the portion of the image have a 
certain color and texture feature? 


15 


10 


What color does the image have at a specified 
position? 




11 


What texture does the image have at a specified 
position? 




12 


What color and texture does the image have at a 
specified position? 



17 
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Table 2 



15 





Query Type 


Main Feature Type 


1 


What color does image have as a 
whole? 


Global color 
information 


2 


Does any portion have a certain 
color feature? 


Local color 
information 


3 


What degree does the portion 
have a certain color feature? 


Local color 
information 


4 


What texture does the image as a 
whole? 


Global texture 
formation 


5 


Does any portion have a certain 
texture feature? 


Local texture 
information 


6 


What degree does the portion 
have a certain texture feature? 


Local texture 
information 


7 


Does the image have a certain 
color and texture feature or 
have any portion having such 
features? 


Global color texture 
information 


a 


Does any portion have a certain 
color and texture feature? 


Local color texture 
information 


9 


What degree does the portion 
have a certain color and texture 
feature? 


local color texture 
information 


10 


What color does the image have 
at a specified position? 


Local color position 
information 


11 


What texture does the image have 
at a specified position? 


local texture 
position information 


12 


What color and texture does the 
image have at a specified 
position? 


local color texture 
poa ition information 



18 
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In Table 2, 12 query types are presented and to satisfy the 
characteristics with respect to such queries, the search syatem 
should have at least the following 8 image feature information. 
The first image feature information is a global color 
5 information which represents the color feature of the whole 
image, A color histogram may be an example of the global color 
information. The second image feature information is a global 
texture information which represents the texture feature of the 
C3 whole image • A texture histogram may be an example of the global 
15) texture information. The feature information of the color and the 
in texture of the whole image may be represented by a combination 
jS of the global color information and the global texture 
information. 

The third image feature information is a local color 

lii information which represents the color feature of a region, i.e. 

C3 grid region, in the image. A representative color for each local 
grid may be an example of the local color information. 
Alternatively, the weights of color elements obtained from the 
global color information may be utilized as the local color 

20 information . 

The fourth image feature element is a local texture 
information representing the texture feature of a grid region in 
the image. A representative texture for each grid may be an 
example of the local texture information. Alternatively, the 

25 weights of texture elements obtained from the global texture 
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information may be utilized as the local texture information. 

The fifth image feature element is a local color and texture 
information which represents the color and texture features of 
a grid region in the image. A representative color and texture 
for each grid may be an example of the local color and texture 
information. Alternatively, the weights of color and texture 
elements respectively obtained from the global color information 
and the global texture information may be utili2ed as the local 
color and texture information. 



^'10 The sixth image feature element is a local color position 



information which represents a color feature in a region at a 
particular position of the image. A color local grid feature may 
J^f be an example of the local color position information. The 

seventh image feature element is a local texture position 
^f5 information which represents a texture feature in a region at a 
particular position of the image. A texture local grid feature 
may be an example of the local texture position information. 
Also, the specified color and texture feature in a region at the 
particular position of the image can be represented as a 
20 combination of the sixth and seventh information. 

The eighth image feature information is a local color and 
texture information which represents a specified color and 
texture feature in a region at a particular position of the 
image. A color and texture local grid feature may be an example 
25 of the local color and texture position information. 
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The system can perform an effective search by constructing 
a set of feature information^ i.e. image characteristics, as 
described above using the analyzed results based upon the 
contents of the queries and add element weights to the 
constructed features. Thus, if a user requests a search, the 
system adjusts the importance of the image characteristics, i.e. 
the weights of the features and/or feature element, and performs 
the image search. 

The search method using a reference multimedia data 
determines a multimedia data having the highest similarity to the 
reference multimedia data by adjusting the weights of the 
features and/or feature elements of the respective features 
included in the multimedia data. Here, the weight adjustment of 
the feature and/or feature elements of the respective features 
can be performed using one of a direct adjusting method by the 
user, an automatic adjusting method by the system, or an 
adjusting method using the relevance information (i.e., positive 
and negative information) fed back to the system by the user. The 
meanings of the features as described above will now be explained 
in detail. 

First, a color histogram represents the color distribution 
in an image. Similarly, the texture histogram represents the 
texture distribution in an image. 

The color image grid represents the color information of a 
grid region generated by dividing an image into n*m grid regions. 



The texture image grid represents the texture information of a 
grid region generated by dividing an image into n*m grid regions. 
The color-texture joint local grid represents the color texture 
information of a grid region generated by dividing an image into 
5 n*m grid regions . 

FIG. 6 shows the structure of texture description which can 
be constructed in consideration of the query types and relevance 
C3 feedbacks of the user 601. The structure comprises comprises a 
=S global information 602 which represents a feature of a whole 
Mb image, a spatial information 603 which represents a feature of 
in an image region, and weight information 604 which represents the 
2 importance of zhe constructed features 602 and 603, The global 

IJ 

M information includes a global feature descriptor 605 of the whole 

5 : 5 

!^ image, and an element weight descriptor 606 of the feature 
Is elements of the global feature descriptor of the whole image. The 
spatial information 603 includes a spatial feature descriptor 607 
of an image region, and a position weight descriptor 608 of the 
image region. 

The global information 602 of the whole image and the 
20 spatial information 603 of the image region can be constructed 
by a selective combination of features included in the image such 
as the color, texture, and shape. Here, the possible combinations 
of the basic features can be obtained using Equation 6 below, 
where n denotes the number of the basic features. 
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Thus, the number of feature types obtained by Equation 6 
applies to local positions and for global information, since 
there n number of basic features, the total number of feature 
5 types can be obtained by Equation 7, 

n 

Q [1] "<^* + 2n • 

=3 

1% The present invention will be explained utilizing two basic 

1% features of color and texture. In such case, the total number of 
feature types required by the system would be 3-i-2*2 = 7, However, 

X 

if the feature of shape is added, the total number of required 
feature type would be 7+2*3=13. 

f — 
f — i 

FIG, 7 shows image characteristics constructed using the 
features of color and texture. Referring to FIG. 7, the relevance 
feedback image (s) 701 used for adjusting the weights of the image 

15 features according to the user feedback includes global color 
information 70?a of the whole image, global texture information 
702b of the whole image, spatial information 703a of image 
regions, spatial color information 703b of image regions, and 
weight descriptor 704 of the global informations 702a and 702b, 

20 and of the spatial informations 703a and 703b. 

In FIG. 7, four feature informations 702a, 702b, 703a and 
703b are used, requiring four weights. Particularly, the global 
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color information 702a includes a global color histogram 705 
representing the color feature information of the whole image, 
and an element weight descriptor 706 of the respective bins of 
the global color histogram. The global texture information 702b 
5 includes a global texture histogram 707 representing the texture 
information of the whole image, and an element weight descriptor 
708 of the respective bins of the global texture histogram. 

^ Also, the spatial color information 703a includes a color 

i 1 

image grid 709, and a position weight descriptor 710 of the color 

%J 

iflo image grid. The spatial texture information 703b includes a 
texture Image grid 711, and a position weight descriptor 712 of 
the texture image grid. 

The color histogram 705 is used as a feature information of 
the whole image and the weight of each color element in the color 
histogram 705 are represented by the element weight descriptor 
706. Also, the global texture histogram 707 is used as another 
feature information of the whole image and the weight of each 
texture element in the global texture histogram. 707 are 
represented by the element weight descriptor 708, 

20 Moreover, the color image grid 709 is used as a feature 

information of the image regions and the weight of each grid 
position in the color image grid 709 is represented by the 
position weight descriptor 710. Similarly, the texture image grid 
711 is used as another feature information of the image regions, 

25 and the weight of each grid position in the texture image grid 
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711 is represented by the position weight descriptor 712. 

As shown in FIG. 1 , an image characteristic structure having 
four feature information was explained in order to satisfy the 
twelve query types in Table 2, However, all nine feature types 
5 for the twelve query types is not necessary. For example, if a 
color-texture joint local grid is used as a feature, the local 
color, local texture, local color and texture, local position 
p color, local position texture and local position color and 
,S texture can be obtained from the color-texture joint local grid. 
fflD Furthermore, in the image characteristic structure of FIG. 

fiJ 

|rj 7, the feature weights are represented the same level as the 

i J 

s feature information, and the feature element weights are 

U represented in a level below the respective feature information. 
U However, image characteristics may be constructed alternatively 
Ss with the feature weights in a level below the respective feature 
information as shown in FIG. 8* For example, assuming that a 
global color information 801 is the feature information, the 
global color information 801 includes a global color feature 802, 
and weights 803. Here the weights 803 is composed of feature 
20 weights 804 corresponding to the global color feature and feature 
element weights 805. 

FIG. 9 shows another embodiment of the image characteristics 
used for adjusting the weights of the image features according 
to the user feedback- In this image characteristic, all 
25 information related to weight characteristics are grouped into 

25 
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a set and represented separately. 

Referring to FIG, 9, the image feature structure 902, i.e. 
the reference feedback, for adjusting the weights of the image 
features when searching the image 901 includes image 
5 characteristics 903 and the weight characteristics 904. The image 
characteristics 903 include global information 905, local 
information 906, and local position information 907. The weight 
,^=. characteristics 904 include feature weights 908 and feature 

element weights 909. Moreover, the global information 905 

J'lo includes n feature units 910, the local information 906 includes 

ry 

a number of feature units 911 equivalent to a sum of the number 
of features and possible combinations of the features, and the 

12 local position information 907 also includes n feature units . 

12 FIG, 10 shows another example of the image data structure 

J^5 of FIG. 9 when the image information includes two basic features 
of color and texture. Particularly, the image characteristics 
1001 includes global information 1002, local information 1003 and 
local positional information 1004. The global information 1002 
includes a global color feature unit 1005 and a global texture 
20 feature unit 1007, The local information 1003 includes a local 
color feature unit 1009, a local texture feature unit 1010 and 
a local color and texture feature unit 1011. The local position 
information 1004 includes a local position color feature unit 
1013 and a local position texture feature unit 1014. 

25 Moreover, the global color feature unit 1005 is represented 
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by a global color histogram 1006, the global texture feature unit 
1007 is represented by a global texture histogram 1008, and the 
local color feature unit 1009 and the local position color 
feature unit 1013 are represented by a color image grid 1012. 
5 Also, the local texture feature unit 1010 and the local position 
texture feature unit 1014 are represented by a texture image grid 
1015. Finally, the local color and texture feature unit IQll is 
represented by both the color image grid 1012 and the texture 
U image grid 1015. 

;=S0 Therefore, the query types in Table 2 can be satisfied by 

in 

J*U constructing image characteristics of the seven features as 
C3 described above, and the weights are updated by adjusting the 

C3 weights in the feature weights and the feature element weights 
iJ as shown in FIG. 11. Referring to FIG, 11, the image feature 

C|5 structure 1102, i,e, the reference feedback/ for adjusting the 
weights of the image features when searching the image 1101 
includes image characteristics 1103 and weight characteristics 
1104. 

Particularly, the image characteristics 1103 includes global 
20 information 1105, local information 1106 and local positional 
information 1107. The weight characteristics 1104 includes 
feature weights 1108 and feature element weights 1109. Here, the 
global information 1105 includes a global color feature unit 1110 
and a global texture feature unit 1111, The local information 
25 1106 includes a Ipcal color feature unit 1112, a local texture 
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feature unit 1113 and a local color and texture feature unit 
1114. The local position information 1107 includes a local 
position color feature unit 1115 and a local position texture 
feature unit 1116. 
5 As described above, according to the present invention, the 

system analyzes all possible queries of the user, and provides 
minimum image characteristics which satisfy all judgement 

[3 standards during the image search. Accordingly, a rapid and 
effective image search can be performed by adjusting the weights 

|H) of the teatures and feature elements to reflect the user 

x'- I 

In feedbacks. 

„ The foregoing embodiments are merely exemplary and are not 

il to be construed as limiting the present invention. The present 
Ji teachings can be readily applied to other types of apparatuses. 
55 The description of the present invention is intended to be 
illustrative, and not to limit the scope of the claims. Many 
alternatives, modifications, and variations will be apparent to 
those skilled in the art. 
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